Implicit Field Solve Preconditioner based on Curl-Curl Operator #5286

debog · 2024-09-18T03:04:54Z

Implemented a preconditioner for the implicit E-field solve using the AMReX curl-curl operator and the MLMG solver.

Introduced a Preconditioner base class that defines the action of a preconditioner for the JFNK algorithm.
Implemented the CurlCurlMLMGPC that uses the multigrid solution for the curl-curl operator (implemented in AMReX) to precondition the E-field JFNK solve.

Other changes needed for this:

Partially implemented a mapping between WarpX field boundary types and AMReX's linear operator boundary types.
Added some functionalities to ImplicitSolver class that allows preconditioners to access WarpX info (like Geometry, boundaries, etc).

Some premilinary wall times for:

Test: inputs_vandb_2d
  Grid: 160 X 160
  dt: 0.125/wpe = 2.22e-18 (dt_CFL = 7.84e-19 s, CFL = 2.83)
  Time iterations: 20

Solver parameters:
  newton.max_iterations = 10
  newton.relative_tolerance = 1.0e-12
  newton.absolute_tolerance = 0.0
  gmres.max_iterations = 1000
  gmres.relative_tolerance = 1.0e-8
  gmres.absolute_tolerance = 0.0

Avg GMRES iterations: ~3 (wPC), ~27 (noPC)

with 32^2 particles per cell:

Lassen (MPI + CUDA)
-------------------
  Box  GPU   Walltime (s)
             wPC       noPC
   1    1    2324.7    15004.1
   4    1    2306.8    14356.8
   4    4     758.9     3647.3

Dane (MPI + OMP)
----------------
  Box  CPU  Threads   Walltime (s)
                      wPC      noPC
   1    1      1      6709.3   43200.0*
   1    1      2      3279.1   22296.1
   1    1      4      1696.3   11613.2
   1    1      8      1085.0    6911.4
   1    1     16       724.3    4729.0
   4    1      1      5525.9   33288.8
  16    1      1      4419.4   28467.8
   4    4      1      1324.4    9121.1
  16   16      1       524.9    3658.8

* 43200.0 seconds is 12 hours (max job duration on Dane);
the simulation was almost done (started the 20th step).

with 10^2 particles per cell:

Lassen (MPI + CUDA)
-------------------
  Box  GPU   Walltime (s)
             wPC       noPC
   1    1    365.0     1443.5 
   4    1    254.1      927.8 
   4    4    133.1      301.5 

Dane (MPI + OMP)
----------------
  Box  CPU  Threads   Walltime (s)
                      wPC      noPC
   1    1      1      440.8    2360.5     
   1    1      2      241.7    1175.8 
   1    1      4      129.3     727.0 
   1    1      8       94.2     407.5 
   1    1     16       74.3     245.6 
   4    1      1      393.3    1932.5 
  16    1      1      337.6    1618.7 
   4    4      1       92.2     479.1 
  16   16      1       58.1     192.6

…l MLMG solver

* Copy rho from guard cells in IGF solver * Reset checksum

Source/NonlinearSolvers/CurlCurlMLMGPC.H

Source/NonlinearSolvers/Preconditioner.H

Source/NonlinearSolvers/CurlCurlMLMGPC.H

…X is not yet implemented in 1D, added precompiler guardrails to not use it in 1D

Source/NonlinearSolvers/JacobianFunctionMF.H

Source/FieldSolver/ImplicitSolvers/ImplicitSolver.cpp

…g new ones and copying back-and-forth

…ng used and clang-tidy complains about them

Source/FieldSolver/ImplicitSolvers/ImplicitSolver.H

Source/NonlinearSolvers/Preconditioner.H

Source/WarpX.H

Source/NonlinearSolvers/CurlCurlMLMGPC.H

Source/NonlinearSolvers/NewtonSolver.H

Source/NonlinearSolvers/CurlCurlMLMGPC.H

Co-authored-by: Weiqun Zhang <[email protected]>

…tion declaration.

JustinRayAngus · 2024-10-29T20:47:29Z

Source/FieldSolver/ImplicitSolvers/ImplicitSolver.H

                              int              a_nl_iter,
                              bool             a_from_jacobian ) = 0;

+    [[nodiscard]] virtual amrex::Real theta () const { return 1.0; }


Making this a member of the base class is fine for now. In a future PR, we can remove it by passing in m_theta*a_dt to the solver. This will require modifications to several files in the ImplicitSolver folder and in the NonlinearSolvers folder, but it will make the code more tidy. I can do this after this PR is merged.

JustinRayAngus

This PR looks good to me.

Implemented implicit field solve preconditioner based on the curl-cur…

ef110d5

…l MLMG solver

debog requested a review from JustinRayAngus September 18, 2024 03:05

debog added enhancement New feature or request component: implicit solvers Anything related to implicit solvers labels Sep 18, 2024

debog and others added 7 commits September 17, 2024 20:07

Merge branch 'development' into implicit_solvers_pc

8196a85

Fixed the paths to header files in subdirectory

248e21e

Docs: how to use add_subdirectory for new tests (BLAST-WarpX#5279)

40d11ab

Copy rho from guard cells in IGF solver (BLAST-WarpX#5284)

6cbebe6

* Copy rho from guard cells in IGF solver * Reset checksum

adding braces to one line logical statements.

0f3218a

initializing m_pc_type = none to prevent crash

daa8edc

Merge branch 'development' into implicit_solvers_pc

ea3838b

JustinRayAngus reviewed Sep 18, 2024

View reviewed changes

Source/NonlinearSolvers/CurlCurlMLMGPC.H Show resolved Hide resolved

JustinRayAngus reviewed Sep 18, 2024

View reviewed changes

Source/NonlinearSolvers/CurlCurlMLMGPC.H Show resolved Hide resolved

JustinRayAngus reviewed Sep 18, 2024

View reviewed changes

Source/NonlinearSolvers/CurlCurlMLMGPC.H Show resolved Hide resolved

JustinRayAngus reviewed Sep 18, 2024

View reviewed changes

Source/NonlinearSolvers/Preconditioner.H Show resolved Hide resolved

debog commented Sep 18, 2024

View reviewed changes

Source/NonlinearSolvers/CurlCurlMLMGPC.H Outdated Show resolved Hide resolved

debog commented Sep 18, 2024

View reviewed changes

Source/NonlinearSolvers/CurlCurlMLMGPC.H Outdated Show resolved Hide resolved

added some comments based on Justin's PR comments

482bc73

debog marked this pull request as ready for review September 19, 2024 19:05

debog requested review from RemiLehe, RevathiJambunathan, WeiqunZhang and ax3l September 19, 2024 19:06

fixed compilation errors in 1D and 3D; since the curl-curl op in AMRe…

aff9965

…X is not yet implemented in 1D, added precompiler guardrails to not use it in 1D

github-advanced-security bot found potential problems Sep 19, 2024

View reviewed changes

Source/NonlinearSolvers/JacobianFunctionMF.H Fixed Show fixed Hide fixed

clang-tidy recommendations

5c45dc1

debog force-pushed the implicit_solvers_pc branch from 7680f9d to 5c45dc1 Compare September 19, 2024 23:11

debog added 2 commits September 19, 2024 17:58

addressed another clang-tidy error

4801d6c

Merge branch 'development' into implicit_solvers_pc

75b88f9

ax3l assigned JustinRayAngus Sep 23, 2024

debog added 2 commits September 24, 2024 11:35

Merge branch 'development' into implicit_solvers_pc

d2d7ea6

updated curl-curl PC to use the ablastr::fields definitions

e601e91

debog commented Sep 27, 2024

View reviewed changes

Source/FieldSolver/ImplicitSolvers/ImplicitSolver.cpp Show resolved Hide resolved

debog added 2 commits September 30, 2024 11:42

added an "else" to the if-loop converting WarpX BC to LinOp BC

79aa72c

Merge branch 'development' into implicit_solvers_pc

0db699d

debog force-pushed the implicit_solvers_pc branch from 5f0b89d to 0db699d Compare September 30, 2024 18:45

debog added 5 commits October 3, 2024 13:43

using aliased multifabs for the curl-curl PC solve instead of creatin…

e9ff4f2

…g new ones and copying back-and-forth

Merge branch 'development' into implicit_solvers_pc

ff0d8d3

removed aliased multifab creation from 1D code since they are not bei…

a698210

…ng used and clang-tidy complains about them

Merge branch 'ECP-WarpX:development' into implicit_solvers_pc

ac479fc

Merge branch 'ECP-WarpX:development' into implicit_solvers_pc

af56e98

WeiqunZhang requested changes Oct 10, 2024

View reviewed changes

debog and others added 5 commits October 10, 2024 11:37

Merge branch 'ECP-WarpX:development' into implicit_solvers_pc

cabcf74

Merge branch 'ECP-WarpX:development' into implicit_solvers_pc

d1bb01b

Update Source/NonlinearSolvers/CurlCurlMLMGPC.H

4228075

Co-authored-by: Weiqun Zhang <[email protected]>

Update Source/NonlinearSolvers/CurlCurlMLMGPC.H

4a3c2ba

Co-authored-by: Weiqun Zhang <[email protected]>

addressed PR comments from Weiqun

70c5f3f

debog requested review from JustinRayAngus and WeiqunZhang October 24, 2024 20:20

debog and others added 3 commits October 25, 2024 11:26

Merge branch 'ECP-WarpX:development' into implicit_solvers_pc

4fe3cfb

Merge branch 'ECP-WarpX:development' into implicit_solvers_pc

0c4c1d7

making function argument names explicit in Precondioner::Apply() func…

42cc9ec

…tion declaration.

JustinRayAngus reviewed Oct 29, 2024

View reviewed changes

JustinRayAngus approved these changes Oct 29, 2024

View reviewed changes

JustinRayAngus requested review from WeiqunZhang and removed request for WeiqunZhang October 30, 2024 05:38

Merge branch 'development' into implicit_solvers_pc

98b0363

WeiqunZhang approved these changes Nov 4, 2024

View reviewed changes

JustinRayAngus merged commit c1cd7ab into BLAST-WarpX:development Nov 4, 2024
37 checks passed

debog mentioned this pull request Feb 14, 2025

Interfacing 1D CurlCurl from AMReX #5662

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implicit Field Solve Preconditioner based on Curl-Curl Operator #5286

Implicit Field Solve Preconditioner based on Curl-Curl Operator #5286

Uh oh!

debog commented Sep 18, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JustinRayAngus Oct 29, 2024

Uh oh!

JustinRayAngus left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Implicit Field Solve Preconditioner based on Curl-Curl Operator #5286

Implicit Field Solve Preconditioner based on Curl-Curl Operator #5286

Uh oh!

Conversation

debog commented Sep 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JustinRayAngus Oct 29, 2024

Choose a reason for hiding this comment

Uh oh!

JustinRayAngus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

debog commented Sep 18, 2024 •

edited

Loading